MPI-2 One-Sided Usage and Implementation for Read Modify Write Operations: A Case Study with HPCC

MPI-2’s one-sided communication interface has become prevalent in scientific applications. One of the important operations in a one sided model is read-modifywrite. MPI-2 semantics provide MPI Put, MPI Get and MPI Accumulate operations which can be used to implement the read-modify-write functionality. The different strategies yield varying performance benefits depending on the underlying one-sided implementation. We use HPCC Random Access benchmark which primarily uses read-modify-write operations as a case study for evaluating the different implementation strategies in this paper. Currently this benchmark is implemented based on MPI two-sided semantics. In this work we design and evaluate MPI-2 versions of the HPCC Random Access benchmark using one-sided operations. To improve the performance, we explore two different optimizations: (i) software based aggregation and (ii) hardware-based atomic operations. We implement aggregation techniques using MPI Accumulate with datatypes to improve the performance of one sided implementation. In order to study the impact of hardware capabilities provided by modern interconnects, we implement a prototype of Accumulate for MPI Sum (Direct Accumulate) using InfiniBand’s atomic fetch and add operation. We evaluate our different approaches on an InfiniBand cluster. We analyze the benefits of software aggregation using datatypes with one-sided operation as well as the hardware based Direct Accumulate. The software based aggregation outperforms the basic one sided scheme without aggregation by a factor of 4.38. The hardware based scheme shows an improvement by a factor of 2.62 as compared to the basic one sided scheme. Our study shows that the software based aggregation performs the best. We also demonstrates the potential and scalability of the hardware based approach. keywords: MPI-2, one-sided, HPCC , Accumulate, InfiniBand